Experiment 2 Confirmatory Performance Analyses
Dave Braun
July 22, 2019
This document is dedicated to conducting the confirmatory performance analyses that were proposed for Experiment 2.
Design and Predictions
Design. The design and analysis is a 2 (difficulty: harder than reference vs. easier than reference) X 2 (difference: moderate vs. extreme) X 2 (transition: repeat vs. switch) within-subjects ANOVA on RTs and error rates.
In retrospect, I’m realizing this design doesn’t make sense because, unlike in Experiment 1, decks aren’t associated with a single level of intensity, but rather the risky deck can always have two outcomes. I think trying to coerce this analysis into the difference X difficulty design would introduce a lot of unnecessary complexity.
I’m taking a step back and thinking about what the actual insights are that I want to get out of this performance data. Are RTs and error rates slower and higher for runs of trials with more switches rather than fewer? Is the switch cost greater when switching occurs less frequently? Thinking about it this way compels me to analyze performance based on numbers of switches in a run, regardless of what deck the run comes from. It might also be interesting to explore how the actual decks influence performance, but that would be more of an exploratory performance analysis.
The analysis I’ll conduct to investigate the above is a regression where number of switches in a run and transition type predict RTs (and errors). The prediction is that RTs will be positively associated with number of switches in a run, RTs will be positively predicted by transition such that switch trials have longer RTs (i.e., the switch cost), and there will be an interaction such that the impact of transition on RT will be greater in runs where switching occurs less frequently.
Results
Response Times
Below is the cleaned data:
d <- read.csv('../../../data/dstClean.csv')
d <- d %>%
filter(transition != 'startBlock')
N <- d %>%
group_by(subject) %>%
summarize(n()) %>%
nrow(.)
dThe sample size is 70.
Visualize the Results
subjectCellMeans <- d %>%
filter(transition != 'startBlock') %>%
group_by(subject, outcomeSwitch, transition) %>%
summarize(rtime = mean(cuedRt))
subjectCellMeans %>%
group_by(outcomeSwitch, transition) %>%
summarize(rt = mean(rtime), se = sd(rtime) / sqrt(N)) %>%
ggplot(aes(x = factor(outcomeSwitch), y = rt, group = transition)) +
geom_point(size = 2) +
geom_line(aes(linetype = transition)) +
geom_errorbar(aes(ymin = rt - se, ymax = rt + se), width = 0.5) +
labs(
x = 'Number of Switches in a Run',
y = 'Response Time (ms)'
) +
scale_linetype_discrete(name = 'Transition Type', labels = c('Repeat', 'Switch')) +
theme_bw() +
theme(legend.position = c(.9,.5),
panel.grid = element_blank())
Fit a linear model
d$outcomeSwitchC <- d$outcomeSwitch - 8
d$transitionE <- ifelse(d$transition == 'repeat', -0.5, 0.5)
m1 <- lm(cuedRt ~ transitionE * outcomeSwitchC, data = d)
plot(m1)
summary(m1)##
## Call:
## lm(formula = cuedRt ~ transitionE * outcomeSwitchC, data = d)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1148.7 -279.3 -106.0 149.6 3737.6
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1020.5843 2.3975 425.686 < 2e-16 ***
## transitionE 269.1011 4.7950 56.121 < 2e-16 ***
## outcomeSwitchC 3.7691 0.7797 4.834 1.34e-06 ***
## transitionE:outcomeSwitchC -6.0660 1.5594 -3.890 1e-04 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 446.4 on 40729 degrees of freedom
## Multiple R-squared: 0.08934, Adjusted R-squared: 0.08928
## F-statistic: 1332 on 3 and 40729 DF, p-value: < 2.2e-16newdata <- data.frame(outcomeSwitch = rep(unique(d$outcomeSwitch), 2), transition = c(rep(-0.5, length(unique(d$outcomeSwitch))), rep(0.5, length(unique(d$outcomeSwitch)))))
newdata$proba <- m1$coefficients[1] + newdata$outcomeSwitch * m1$coefficients[3] + newdata$transition * m1$coefficients[2] + newdata$transition * newdata$outcomeSwitch * m1$coefficients[4]
newdata$transitionE <- newdata$transition
newdata$outcomeSwitchC <- newdata$outcomeSwitch - 8
newdata$transition <- ifelse(newdata$transition == -0.5, 'repeat', 'switch')
newdata <- cbind(newdata, predict(m1, newdata, interval = 'predict'))
ggplot(newdata, aes(x = factor(outcomeSwitch), y = fit, group = transition)) +
geom_jitter(data = subjectCellMeans, aes(x = factor(outcomeSwitch), y = rtime, color = factor(transition)), alpha = .2, width = .2, height = 0, show.legend = FALSE) +
geom_line(size = 2, aes(color = factor(transition))) +
#geom_ribbon(aes(ymin = lwr, ymax = upr, fill = factor(transition)), alpha = .1, show.legend = FALSE) +
scale_color_manual(name = 'Transition Type', values = c(`repeat` = 'blue', `switch` = 'red'), labels = c('Repeat', 'Switch')) +
#scale_fill_manual(values = c(`repeat` = 'blue', `switch` = 'red')) +
labs(
x = 'Number of Switches in a Run',
y = 'Response Time (ms)',
caption = 'Lines reflect predictions from regression. Points reflect subject-wise cell means.'
) +
theme_bw() +
theme(legend.position = c(.8,.85),
panel.grid = element_blank()) 
Centering at high number of switches per run (12)
d$outcomeSwitchC <- d$outcomeSwitch - 12
d$transitionE <- ifelse(d$transition == 'repeat', -0.5, 0.5)
m1 <- lm(cuedRt ~ transitionE * outcomeSwitchC, data = d)
summary(m1)##
## Call:
## lm(formula = cuedRt ~ transitionE * outcomeSwitchC, data = d)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1148.7 -279.3 -106.0 149.6 3737.6
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1035.6608 3.9708 260.817 < 2e-16 ***
## transitionE 244.8371 7.9417 30.829 < 2e-16 ***
## outcomeSwitchC 3.7691 0.7797 4.834 1.34e-06 ***
## transitionE:outcomeSwitchC -6.0660 1.5594 -3.890 1e-04 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 446.4 on 40729 degrees of freedom
## Multiple R-squared: 0.08934, Adjusted R-squared: 0.08928
## F-statistic: 1332 on 3 and 40729 DF, p-value: < 2.2e-16And centering at low number of switches per run (4)
d$outcomeSwitchC <- d$outcomeSwitch - 4
d$transitionE <- ifelse(d$transition == 'repeat', -0.5, 0.5)
m1 <- lm(cuedRt ~ transitionE * outcomeSwitchC, data = d)
summary(m1)##
## Call:
## lm(formula = cuedRt ~ transitionE * outcomeSwitchC, data = d)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1148.7 -279.3 -106.0 149.6 3737.6
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1005.5079 3.8965 258.057 < 2e-16 ***
## transitionE 293.3652 7.7929 37.645 < 2e-16 ***
## outcomeSwitchC 3.7691 0.7797 4.834 1.34e-06 ***
## transitionE:outcomeSwitchC -6.0660 1.5594 -3.890 1e-04 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 446.4 on 40729 degrees of freedom
## Multiple R-squared: 0.08934, Adjusted R-squared: 0.08928
## F-statistic: 1332 on 3 and 40729 DF, p-value: < 2.2e-16Looking at the relationship between switches in a run and mean RT (dropping transition)
m2 <- lm(cuedRt ~ outcomeSwitch, data = d)
slope <- round(m2$coefficients[2], 2)
summary(m2)##
## Call:
## lm(formula = cuedRt ~ outcomeSwitch, data = d)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1088.0 -306.8 -108.8 172.6 3711.9
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 850.327 6.349 133.94 <2e-16 ***
## outcomeSwitch 20.622 0.747 27.61 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 463.4 on 40731 degrees of freedom
## Multiple R-squared: 0.01837, Adjusted R-squared: 0.01834
## F-statistic: 762 on 1 and 40731 DF, p-value: < 2.2e-16sMeans <- d %>%
group_by(subject, outcomeSwitch) %>%
summarize(rtime = mean(cuedRt))
sMeans %>%
group_by(outcomeSwitch) %>%
summarize(rt = mean(rtime), se = sd(rtime) / sqrt(N)) %>%
ggplot(aes(x = factor(outcomeSwitch), y = rt, group = 1)) +
geom_line() +
geom_point(size = 2.5) +
geom_jitter(data = sMeans, aes(x = factor(outcomeSwitch), y = rtime), alpha = .2, width = .1, height = 0) +
geom_errorbar(aes(ymin = rt - se, ymax = rt + se), width = 0.5) +
annotate('text', x = 4, y = 1500, label = paste('b = ', slope, sep = '')) +
labs(
x = 'Number of Switches in a Run',
y = 'Response Time (ms)'
) +
theme_bw() +
theme(panel.grid = element_blank())
Error Rates
Below is the cleaned data:
d <- read.csv('../../../data/dstCleanErrors.csv')
d <- d %>%
filter(d$transition != 'startBlock')
N <- d %>%
group_by(subject) %>%
summarize(n()) %>%
nrow(.)
dThe sample size is 70.
Visualize the Results
subjectCellMeans <- d %>%
group_by(subject, outcomeSwitch, transition) %>%
summarize(error = mean(error))
subjectCellMeans %>%
group_by(outcomeSwitch, transition) %>%
summarize(err = mean(error), se = sd(error) / sqrt(N)) %>%
ggplot(aes(x = factor(outcomeSwitch), y = err, group = transition)) +
geom_point(size = 2) +
geom_line(aes(linetype = transition)) +
geom_errorbar(aes(ymin = err - se, ymax = err + se), width = 0.5) +
labs(
x = 'Number of Switches in a Run',
y = 'Error Rate'
) +
scale_linetype_discrete(name = 'Transition Type', labels = c('Repeat', 'Switch')) +
theme_bw() +
theme(legend.position = c(.2,.8),
panel.grid = element_blank())
Fit a linear model
d$outcomeSwitchC <- d$outcomeSwitch - 8
d$transitionE <- ifelse(d$transition == 'repeat', -0.5, 0.5)
m1 <- lm(error ~ transitionE * outcomeSwitchC, data = d)
plot(m1)
summary(m1)##
## Call:
## lm(formula = error ~ transitionE * outcomeSwitchC, data = d)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.04543 -0.04254 -0.02880 -0.02490 0.97823
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.378e-02 9.234e-04 36.577 < 2e-16 ***
## transitionE 1.463e-02 1.847e-03 7.922 2.38e-15 ***
## outcomeSwitchC 7.523e-04 2.997e-04 2.510 0.0121 *
## transitionE:outcomeSwitchC -5.761e-05 5.994e-04 -0.096 0.9234
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.1803 on 44750 degrees of freedom
## Multiple R-squared: 0.00221, Adjusted R-squared: 0.002143
## F-statistic: 33.04 on 3 and 44750 DF, p-value: < 2.2e-16newdata <- data.frame(outcomeSwitch = rep(unique(d$outcomeSwitch), 2), transition = c(rep(-0.5, length(unique(d$outcomeSwitch))), rep(0.5, length(unique(d$outcomeSwitch)))))
newdata$proba <- m1$coefficients[1] + newdata$outcomeSwitch * m1$coefficients[3] + newdata$transition * m1$coefficients[2] + newdata$transition * newdata$outcomeSwitch * m1$coefficients[4]
newdata$transitionE <- newdata$transition
newdata$outcomeSwitchC <- newdata$outcomeSwitch - 8
newdata$transition <- ifelse(newdata$transition == -0.5, 'repeat', 'switch')
newdata <- cbind(newdata, predict(m1, newdata, interval = 'predict'))
ggplot(newdata, aes(x = factor(outcomeSwitch), y = fit, group = transition)) +
geom_jitter(data = subjectCellMeans, aes(x = factor(outcomeSwitch), y = error, color = factor(transition)), alpha = .2, width = .2, height = 0, show.legend = FALSE) +
geom_line(size = 2, aes(color = factor(transition))) +
#geom_ribbon(aes(ymin = lwr, ymax = upr, fill = factor(transition)), alpha = .1, show.legend = FALSE) +
scale_color_manual(name = 'Transition Type', values = c(`repeat` = 'blue', `switch` = 'red'), labels = c('Repeat', 'Switch')) +
#scale_fill_manual(values = c(`repeat` = 'blue', `switch` = 'red')) +
labs(
x = 'Number of Switches in a Run',
y = 'Error Rate',
caption = 'Lines reflect predictions from regression. Points reflect subject-wise cell means.'
) +
ylim(0,.2) +
theme_bw() +
theme(legend.position = c(.8,.85),
panel.grid = element_blank()) ## Warning: Removed 26 rows containing missing values (geom_point).
Looking at the relationship between switches in a run and mean RT (dropping transition)
m2 <- lm(error ~ outcomeSwitch, data = d)
slope <- round(m2$coefficients[2], 5)
summary(m2)##
## Call:
## lm(formula = error ~ outcomeSwitch, data = d)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.04374 -0.03707 -0.03374 -0.02874 0.97626
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.0204104 0.0023651 8.630 < 2e-16 ***
## outcomeSwitch 0.0016662 0.0002767 6.022 1.74e-09 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.1804 on 44752 degrees of freedom
## Multiple R-squared: 0.0008097, Adjusted R-squared: 0.0007874
## F-statistic: 36.26 on 1 and 44752 DF, p-value: 1.736e-09sMeans <- d %>%
group_by(subject, outcomeSwitch) %>%
summarize(error = mean(error))
sMeans %>%
group_by(outcomeSwitch) %>%
summarize(err = mean(error), se = sd(error) / sqrt(N)) %>%
ggplot(aes(x = factor(outcomeSwitch), y = err, group = 1)) +
geom_line() +
geom_point(size = 2.5) +
geom_jitter(data = sMeans, aes(x = factor(outcomeSwitch), y = error), alpha = .2, width = .1, height = 0) +
geom_errorbar(aes(ymin = err - se, ymax = err + se), width = 0.5) +
annotate('text', x = 4, y = .3, label = paste('b = ', slope, sep = '')) +
labs(
x = 'Number of Switches in a Run',
y = 'Error Rate'
) +
theme_bw() +
theme(panel.grid = element_blank())
A work by Dave Braun
dab414@lehigh.edu